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THE EFFECTS OF CREW RESOURCE MANAGEMENT (CRM) TRAINING 

IN AIRLINE MAINTENANCE: 

RESULTS FOLLOWING THREE YEAR’S EXPERIENCE 
Report of Work Completed through 12-31-94 


SUMMARY 


This report describes three year's evaluation of the effects of one airline's Crew 
Resource Management (CRM) training operation for maintenance. This evaluation 
focuses on the post-training attitudes of maintenance managers' and technical support 
professionals, their reported behaviors, and the safety, efficiency and dependable 
maintenance performance of their units. The results reveal a strong positive effect of the 
training. 


The overall program represents the use of CRM training as a long-term 
commitment to improving performance through effective communication at all levels in 
airline maintenance operations. The initial findings described in our previous progress 
reports are reinforced and elaborated here The current results benefit from using the 
final pre-post training survey population , with a total of over 2,000 managers and staff 
professionals. Additionally there are now full results from the two-month, six-month, and 
12-month follow-up questionnaires, together with as many as 33 months of post-training 
performance data, using several indicators. 


In this present report, we examine participants’ attitudes, their reported behaviors 
following the training, the performance of their work units, and the relationships among 
these variables. Attitudes include those measured immediately before and after the 
training as well as attitudes of the participants months after their training. Performance 
includes measures, by work units, of on-time flight departures, on-schedule maintenance 
releases, occupational and aircraft safety, and efficient labor costs. We report changes in 
these performance measures following training, as well their relationships with the training 
participants’ attitudes. 
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Highlights of results from this training program include increased safety and 
improved costs associated with positive attitudes about the use of more assertive 
communication, and the improved management of stress. Improved on-time performance 
is also related to those improved attitudes, as well as favorable attitudes about 
participative management. More specific results are as follows: 


1 ) We note positive trends in a number of the company's overall maintenance 
performance indicators for the months after the onset of training (variously 27 to 34 
months, depending on the measure), compared with the months before (varying from 5 to 
17 by measure). Measures in the Safety Performance category improve in the 32 months 
measured after the training. In addition, results in the Dependability Performance category 
continue to improve in the 27 months measured after the onset of training. The available 
measure for Efficiency also shows generally effective performance during the first 1 8 
months following training, but then performance slips for the remaining 16 months 
measured. 


2) Comparisons of managers' attitudes immediately after their training, with their 
pre-training attitudes, show significant improvement for most of those expected. 
Improvement is noted in attitudes about "willingness to share command responsibility," 
"usefulness of communication & coordination," and "recognition that stressors effect 
management decision making." The same attitudes of these managers two-, six-, and 12- 
months later, reveal that these favorable post-training attitudes remain at those high levels 
in the months after the training The fourth attitude scale measured, “willingness to voice 
disagreement” (a measure of assertiveness), although showing no significant change 
immediately following training, improved significantly above the pre-training levels two 
months after training and it remained at that higher level six and 12 months afterwards. 
The influence of the training on all the expected participant attitudes is thus a stable and 
robust change, and not merely a brief "honeymoon effect.” 


3) Other analyses of the data look at the relationships between pre-, and post- 
training attitudes and maintenance performance over several years. In this "time-lagged" 
correlation analysis we tested the effects for performance before the onset of the training 
program, immediately after the training program and for subsequent effects on 
performance months after the training and/or follow-up surveys began. 


We found a pattern of significant relationships between the participants’ pre- 
training and post -training attitudes and performance. Attitudes about skills and behaviors 
specifically taught in the CRM training were predictive of subsequent maintenance 
performance. Attitudes, immediately following training, about the value of 
communication & coordination, as well as about the “value of stress management,” were 
found to be related to on-time scheduled maintenance performance both prior to and after 
the training began. 


2 



The strongest effects of CRM training appear on respondent attitudes two months 
after their training. Examination of that two-month follow up survey reveals that attitudes 
about sharing command responsibility, and about assertiveness (both of these are "active" 
beliefs emphasized in the training, and which require behavioral support to reinforce 
them), are related to subsequent on-time flight departures, and occupational safety. 
Attitude about sharing responsibility is also related to many months of timely maintenance 
work both before and after the training began Overall, the correlations between 
performance and the attitudes measured in this two-month follow-up survey are 
remarkably strong and indicate that lessons learned from the CRM training have 
consolidated after two months, and in "active" ways. 


The six-month follow-up results show a retreat from the effects in the 2-month 
survey. The 6-month results reveal that only prior maintenance performance is related to 
surveyed opinions about delegation, communication and stress management. 


By the 12-month follow-up survey, the CRM training effects have returned to 
mirror aspects of the post and 2-month surveys. Results of this later survey suggest that 
the lapse at 6 months probably represents a consolidation of the training effects rather than 
their elimination. Training participants continued to develop the skills they learned in 
CRM and used them to good effect Attitudes about the value of assertiveness one year 
after CRM training were related to many months of occupational safety both before and 
after that survey was taken. Attitudes, a year after training, toward sharing command 
decisions were (reminiscent of the 2 month results) also related to many months of 
occupational safety. Finally, 12-month attitudes about managing stress effects were 
associated with improved performance in the very stressful performance areas of meeting 
ambitious goals for cost saving, and for timely maintenance These findings strongly 
suggest that lessons of stress management, learned during training more than a year 
earlier, had also developed and matured for the training participants. 
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THE EFFECTS OF CREW RESOURCE MANAGEMENT (CRM) TRAINING 

IN AIRLINE MAINTENANCE: 

RESULTS FOLLOWING THREE YEAR’S EXPERIENCE 


J.C. Taylor Ph.D. & M.M. Robertson Ph.D. 
Institute of Safety and Systems Management 
The University of Southern California 
Los Angeles, December 1994 


THE STUDY 


Background 

The CRM program reported here involves initial training in several team-related 
concepts, including communication skills, self-knowledge, situational awareness, and 
assertiveness skills for maintenance management and support staff personnel in one large 
U.S. airline (hereinafter called "the company"). Maintenance (or "Technical Operations") 
in this company includes engineering, quality assurance, technical planning, systems & 
procedures, contracts administration and purchasing, as well as the more direct 
maintenance functions of line- and base-maintenance, inspection, shops and material 
services The company succeeded in training all of its maintenance directors, managers, 
supervisors and assistant supervisors, engineers, planners, coordinators, and schedulers, as 
well as a large part of its inspection staff. According to company records, 2, 199 people in 
total completed the CRM for Maintenance course between June 1991 and March 1994. 

CRM training in airline maintenance operations was highly unusual when 
introduced into the present organization. This exceptional example has been intensively 
studied over a three year period The first papers ( cf , Taylor, 1991a, 1991b) reported the 
methodological and measurement characteristics of the attitude and performance 
indicators used. Subsequent reports have been limited to a relatively small sample of 
participants, and a small number of months of maintenance performance data subsequent 
to the training. A summary report of the first six months experience (prepared by Taylor, 
Bettencourt & Robertson, at the University of Southern California in 1992) was published 
in the FAA Office of Aviation Medicine's second annual human factors R&D progress 
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report (Galaxy, 1993); and a subsequent report on the first full year’s experience (also 
prepared at the University of Southern California in 1993) has been included in the FAA 
Office of Aviation Medicine's third progress report (Galaxy, 1994). Several shorter papers 
dealing with selected topics from this study have also been published (Stelly & Taylor, 
1992; Taylor, Robertson, Peck, & Stelly, 1993, Robertson, Taylor, Stelly, & Wagner, In 
Press). The present report concludes the evaluation of the company’s initial program, by 
documenting the mutual effects between training-related attitudes and behaviors and work 
unit performance, as well as the persistence and stability of the attitude and behavioral 
changes following the training. 

The effectiveness of this training, as measured by its ongoing evaluation, can help 
direct both the NASA and airline industry's maintenance human resources practices in the 
future and guide the development of future ATA and FAA training policies and 
regulations. 

The analyses reported below assess the relationships among respondents' training- 
induced attitudes about a variety of management and organizational items, their reported 
post-training behaviors, and maintenance unit performance. 


The Purpose of the Program and of the Course. 

The program's champion was the company's Senior Vice President for Technical 
Operations. He announced that his aim for the joint training and evaluation program was 
to improve human resource (HR) management by using science-based tools and 
techniques for evaluating the training outcomes and using those results for continuously 
improving the program's effectiveness (Fotos, 1991). 

This particular training program originally began with advice and assistance from 
the company's flight operations training group who had nearly a decade's experience with 
their own CRM program. After that initial assistance, the company’s flight operations and 
technical operations (i.e., maintenance) CRM programs were directed and administered 
separately. During the first three years of that maintenance CRM program it continued to 
be managed and administered by maintenance personnel, and the trainers were mainly 
maintenance people too. 

The purpose of the training, as reiterated by trainers on the first day of each 
training session, was "To equip all maintenance personnel (management first) with the 
skill to use all resources to improve safety and efficiency." 

Course objectives. The objectives (the more specific goals of the training) were 
also clearly stated during the trainers' introductory remarks: 

1) Diagnose organizational "norms" and their effect on safety. 

2) Promote assertive behavior. 
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3) Understand individual leadership styles 

4) Understand and manage stress 

5) Enhance rational problem solving and decision making skills 

6) Enhance interpersonal skills 

The course was designed for the objectives. The aims and objectives of the 
training were achieved by following a course syllabus containing 12 modules (Appendix A 
contains the syllabus). 

The first phase of training completed. By the conclusion of this phase of the CRM 
training intervention, in February, 1994, nearly 2,200 participants had attended the course. 
Called “the first phase” of training, it included all Technical Operations management and 
staff professionals, and it is the subject of this present report. During early 1994, the usual 
number of new hires or recent promotions to management ranks continued to attend the 
CRM course and those sessions were also attended, for a few months, by Aircraft 
Inspectors, as the beginning of a second phase which would include all mechanics and 
inspectors in the training The present analysis uses data from the entire sample of 
maintenance management, salaried professional staff, and inspectors (n=2,199) who 
participated in the “first phase” training course between June 1991 and March 1994 

Individual respondents as the focus of analyses. To explore some of the effects of 
the training on all individuals, the data from all training participants will be used. The 
remaining (and majority of those) analyses, however, examine the attitudes of respondents 
who (through the use of their confidential code numbers) can be matched between the pre- 
training survey and the later ones. 

Maintenance work units as the focus of analyses. The maintenance performance 
data (classified into categories of "safety," "dependability," and "efficiency") were 
measured by work units, not by individual respondents. The analyses described in this 
report illustrate the effect of changes in respondent attitudes upon the maintenance 
performance of their work-units. For managers these are the units they lead, and for staff 
professionals, maintenance foremen, and inspectors these units are the stations and 
locations to which they belong. In order to accomplish the examination of attitudes 
correlated with performance, the individual respondent’s attitude data were combined into 
averages for the appropriate units. 


The "Crew Resources Mgt/Tech Operations Questionnaire" (CRM/TOQ) 

Prior experience in measurement of attitudes related to CRM training. The 
Cockpit Management Attitudes Questionnaire (CMAQ) has long been a recognized 
measure for assessing flight crew attitudes. It is useful as a training, evaluation and 
research tool ( cf . , Helmreich, Foushee, Benson, & Russini, 1986). The 1990 CMAQ 
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questionnaire contained 25 items measuring attitudes that are either conceptually or 
empirically related to CRM. Taggart (1990) revised the CMAQ for use in a maintenance 
department, and reported positive initial results following CRM training conducted for 
maintenance managers in late 1989. 

Two previous studies have explored the CMAQ instrument for a consistent 
internal structure using the Factor Analysis technique (Gregorich, Helmreich, & Wilhelm, 
1990; Sherman, 1992). In these two studies, using samples of flight crews and air traffic 
controllers, the authors showed that the relationships among the 25 CMAQ items 
clustered into the following four constellations of attitudes: 

1 ) Sharing Command Responsibility, 

2) Value of Communication & Coordination, 

3) Recognizing and Managing Stressor Effects, 

4) Avoidance of Interpersonal Conflict 

Gregorich, et al. (1990) eventually reduced their set from four to three composites 
by dropping "Avoidance of Interpersonal Conflict." Sherman, on the other hand, found 
that fourth factor to be much more robust in his sample of Air Traffic Controllers, and he 
titled it "Advocacy and Assertiveness." Those authors combined the individual CMAQ 
items into three or four composite index scales, respectively, to obtain more stable 
indicators of underlying concepts. Such indices permit a more detailed assessment of the 
separate but related attitudes than a single total score for the entire questionnaire, but they 
also provide more accurate and reliable results than are available from each of the 
individual questionnaire items alone. 

Measurement of attitudes in the present study. The "Crew Resources 
Management/Technical Operations Questionnaire" (CRM/TOQ) developed for the present 
study is a modified version of Taggart's revised CMAQ. In choosing to draw on the 
CMAQ to obtain a "good" measure of the complex concepts of CRM training, the - 
company and the researchers wished to start with a survey, already proven valid as an 
measure of CRM training in the airline industry The two parties also wanted more 
maintenance related measures than the CMAQ provided — in particular they wished to 
measure opinions, important in maintenance, but which were not expected to change 
following the training; as well as rather more specific expectations for future use, as well 
as reports of how the training was subsequently used. The CRM/TOQ followed the 
successful format set by the CMAQ, in that it is a short questionnaire, yet having enough 
items to provide convergence to a smaller, easily used, set of concepts. 

The CRM/TOQ contains 26 multiple response items. The company's modifications 
of the CMAQ involved removing five questions and adding six others. The five questions 
were removed because they either lacked predictive validity as reported by earlier flight 
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crew studies (Helmreich, et al 1986) or, in the company's opinion, lacked relevance to 
maintenance. 

A confirmatory Factor Analysis was undertaken for the data obtained with the 
CRM/TOQ (Taylor, 1991a). Results for the items drawn from the revised CMAQ were 
similar to those of Gregorich, et al. (1990), and subsequently with Sherman (1992). As in 
Sherman’s study, the CRM/TOQ's fourth composite was statistically strong, and was 
therefore retained as the reflected index "Willingness to Voice Disagreement." 

Six questions were also added to the CRM/TOQ, based on items intended to 
measure respondents' perceptions of behaviors dealing with setting and attainment of work 
goals (Geirland & Cotter, 1990). These six individual items were considered important to 
add because the work of maintenance managers differs from that of Flight Operations’ 
officers (as managers) in the typically longer time required for Technical Operation's goal 
attainment and the relatively greater ambiguity of those goals. In addition, because goal 
setting and attainment were not covered in the CRM training such items could act as 
"control questions" about events in the respondents’ work lives which were not predicted 
to change in a consistent or positive way following the training. These questionnaire items 
were separately tested by Factor Analysis and results suggested a two factor structure: 
"Goal sharing in one’s own group," and "Goal sharing with other groups." 

Table 1 displays the six scales calculated from the CRM/TOQ and which are used 
to test the effects of the CRM training on individual participants. Table 1 presents those 
six scales and the numbers of the individual item used to calculate each one. The 
individual item numbers correspond to the numbering of the sample questionnaire located 
in Appendix B. 


TABLE 1 


Scales Used to Test Individual Effects of CRM Training 


SCALE 

CONSTITUENT ITEMS (See Appendix B) 

Scales Sensitive to CRM 

Sharing Command Responsibility 

6, 8, 11, 13, 19 (reflected) 

Communication & Coordination 

5, 12, 14, 16, 17 

Managing Stress 

9, 20 

Assertiveness 

1 , 2 (reflected) 


" Control Scales ’’ 


Sharing Goals in own Group 

21, 22, 23, 24 

Sharing Goals with other Groups 

25, 26 
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Measurement Characteristics of the CRM/TOO. Both reliability and validity of the 

separate items and the composite scales were tested, and they demonstrated good 

measurement qualities (Taylor, 1991a). 

Four versions of the CRM/TOO. There are four versions of the CRM/TOQ 
questionnaire which were used in various phases of this project. 

1. A "Baseline Questionnaire" was mailed to all 1,800 maintenance managers, 

supervisors, and assistant supervisors in the company in May 1991, before the 
training was announced. The results of this baseline assessment of management 
attitudes were used in the Factor Analysis described above and were also used to 
establish the reliability and validity of the CRM/TOQ (Taylor, 1991a). The 
Baseline data were also compared with the subsequent “Pre-training” survey and 
the comparable questions’ mean scores were found to be virtually indistinguishable 
from one another (Taylor, Bettencourt & Robertson [Galaxy], 1993). The 
baseline survey played no further role in the present study. The first 26 multiple- 
choice questions were used (with only minor variations) for all subsequent versions 
of the CRM/TOQ. 

2. A "pre-training" questionnaire was completed by all participants immediately before 

each workshop began. Respondents were asked to choose a private identification 
(I.D.) or code number and to write it on their questionnaire. It was explained that 
this number would allow their responses to be compared with their responses on 
subsequent, follow up questionnaires, but without identifying them. These pre- 
training attitudes were subsequently compared with attitudes immediately after the 
training, as well as with attitudes measured months later. 

3. A "post-training" questionnaire was completed by participants at each workshop's 

conclusion Respondents, again, wrote their I.D. code on the questionnaire; and 
were asked to note that number in their course workbook to help them remember 
it for later "follow-up" questionnaires. The training facilitators collected and 
mailed the completed pre-training and post-training questionnaires to the 
University for processing. Data from this post-training version of the CRM/TOQ 
were compared with the pre-training questionnaire data. As described further 
below, three attitude scales had statistically higher mean scores immediately 
following training. The fourth attitude, “assertiveness,” was not statistically higher 
immediately after training. The two opinion scales about goal sharing were not 
statistically different before and after the training. 

4. "Follow-up" questionnaires were individually mailed by the company to all past 

participants two, six, and twelve months following their CRM training. 
Respondents were again asked to include their private I.D. code on their 
questionnaires, so that those follow-up surveys could be matched with the earlier 
questionnaires. They were provided an pre-addressed and stamped envelope with 
which to return the questionnaires to the University for processing. Although the 
2, 6, and 12 month follow-up surveys were all identical in form, they measured the 
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respondents' thoughts, assessments, and attitudes over increasingly lengthy periods 
from the training (an example of the CRM/TOQ Follow-up questionnaire is 
included in Appendix B). 


Questionnaire return rates. Table 2 presents the return rates for the five surveys 
reported here. The total number of surveys returned and the percentage returned are 
shown for each survey. 


TABLE 2 

Sample Size And Response Ratios: Five Surveys 
CRM/TOQ Questionnaires Received by October, 1994 (2,199) total participants) 


SURVEY 

TOTAL RETURN 

RETURN RATE 

Pre-training Survey 

2,056 

93% 

Post-training Survey 

2,053 

93% 

2-month Follow-up Survey 

810 

37% 

6-month Follow-up Survey 

722 

33% 

12-month Follow-up Survey 

540 

25% 


The high return rate for the pre and post-training surveys result from participants 
being asked to complete the questionnaires while they were present in the training 
sessions. Lower return rates for the subsequent follow-up surveys is the normal result of 
surveys conducted by mail, but the 25%-37% return rates displayed in Table 2 are lower 
than desired. During the first year of the CRM program, we reported that the follow-up 
survey return rates ranged from 40%-45% (Taylor, Bettencourt & Robertson [Galaxy], 
1993). During the intervening two years, although maintenance management continued to 
fully support their CRM program, the company experienced considerable turbulence in its 
markets and it reacted with several waves of staff reductions and station changes 
(including closures in several cases). These changes introduced consequent personnel 
moves which made it extremely difficult for the CRM course administrators to 
subsequently locate course participants to send them surveys. The lower follow-up survey 
return rates for the entire three years reflect this administrative difficulty — using out of 
date or incorrect address lists caused many questionnaires not to be sent to their intended 
recipients. Although lower morale due to layoffs and station closures can be hypothesized 
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to contribute to a lower return rate, the stable attitudes measured in the considerable 
number of follow-up questionnaires received do not reflect it. 

Testing the effects of missing data on scale scores. In any survey instrument like 
the CRM/TOQ, individual respondents will occasionally make a mistake or omit an 
answer to a specific question. Such errors and omissions result in “missing data” 

(unusable answer, or no answer given). When individual items with missing data must be 
used with others with valid responses to form scales, such as we have done for the present 
study, the research investigator faces the choice of dealing with missing data for individual 
questions in three ways. The investigator: 1) can eliminate the scale in question for that 
respondent entirely, or 2) can calculate the scale score without the individual item 
included, or 3) can substitute the population mean score for that item to replace that 
respondent’s “missing data.” Each approach has advantages. The first and second 
alternatives use only actual data, but the former reduces the sample size, and the latter 
biases the scale in favor of the remaining items. The third alternative maintains the sample 
size with a minimum distortion of the resulting score for the total sample. 

We compared both the first and the third alternatives and examined the increase in 
sample size by replacing missing responses in individual items with the sample mean before 
calculating the scale score. There are four attitude scales, and two “control scales” (cf. 
Table 1), measured for both the pre and post-training administrations of the CRM/TOQ 
questionnaire, for a total of 12 modified scale mean scores. The increase in total sample 
“n” using the third alternative modification was between 50 and 255, depending on the 
particular scale in question. As noted above, the total number of questionnaires returned 
was, 2,056 for pre-training, and 2,053 for post-training samples (cf.. Table 2). The 
smallest “n’s” using the unmodified scales were 1,801 and 1,804 for the pre-training “goal 
sharing” scales. After replacing the missing data for every individual item used to 
construct a scale the obtained “n” for every scale in the pretest and posttest was returned 
to 2,056 and 2,053 respectively. 

We also compared the the third alternative modifications with unmodified scales to 
test the degree of possible distortion in scores introduced by the former. Mean values for 
the 12 scales, modified to correct for missing data, were compared with the 12 unmodified 
ones. Eleven of the 12 scales so modified were unchanged from the original (unmodified) 
data set. Only one of the “control scales,” "My unit shares goals with other work units in 
Technical Operations, " was considerably higher in its modified form for the pre-test 
sample (0.30 scale point higher on a five-point scale). The modified post-test score for 
this scale was undifferentiated from the unmodified version. For 1 1 of the 12 items tested, 
using the population mean score to replace missing data was a net benefit — it increased 
the sample size without affecting the scores themselves 2 . 


2 The question of the outlier, however, still remains. Why should this one scale behave so differently from 
the rest? At least a part of the answer is that a number of senior administrative and management people 
reported very positive contact with other groups and, coincidentally, they also did not answer all the 
questions on that topic. The original data collected during the first few months of the program (June- 
Aug., 1991) showed a tendency to over-evaluate the items comprising this scale (cf., Taylor 1991a). A 
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Because replacing missing data with population means had no effect on 1 1 of the 
12 scales so treated we used that modification for in all data analyses in the present report. 


Testing CRM/TOO questionnaire sensitivity. The quantitative questions in the 
CRM/TOQ were designed to obtain four independent attitude scales ( cf , Table 1) and 
several individual opinion items, all specifically sensitive to the CRM training curriculum. 
There were also two “control” scales, dealing with perception of goal setting and 
attainment, which were specifically intended not to be sensitive to the training. As noted 
in the preceding section, one of these control scales was the only one of the six which was 
affected by the scale modification procedure to correct for missing data. Table 3 presents 
additional evidence for the differences between the CRM attitude scales and the control 
scales. Table 3 is a intercorrelation matrix between the four attitude and the two control 
scales from the pre-training survey and those six scales from the post-training survey. The 
cell entries are Pearson Product Moment Correlation coefficients (“r”) and those in italics 
are statistically significant at the .05 confidence level or higher. Boldface coefficients are 
those above r= 49, representing unusually high correlations, — those which account for at 
least 25% of the joint variance of the two correlated scales. That would mean that about 
25% of a given attitude scale’s fluctuation was explained by the fluctuation of the other 
scale in correlation with it. 


disproportionate number of respondents during those first few months of training were senior executives 
and managers. These respondents, as a group, probably feel more certain in their views, and thus would 
be expected to have a lower missing data rate, and would also be more likely to have an answer to 
questions about “other groups” in Technical Operations. After that initial few months, the pre-training 
mean scores for that scale, settled to a lower level which remained unchanged from the post-training 
scores for the remainder of the nearly three years we measured. The individual questions dealing with 
goal sharing with other groups had a higher missing data rate (were answered less often) than all other 
multiple response questions. In the later training sessions an increasing number of respondents did not 
answer one or both of the questions asked about the topic. Later respondents probably felt less certain 
about this concept and those source behaviors. In the final sample there are 252 (roughly 12% of the 
population) pre-training subjects who did not answer one or both of the questions comprising the scale, 
"My unit shares goals with other work units in Technical Operations." When we examined the 
differences across the various departments in Tech Operations, we found that pre-training modified scale 
scores for the mechanical trades departments (line, base, and shop maintenance) were largely unchanged 
from their original data. However, personnel in the other Technical Operations departments (many of 
them higher managers) show ed marked increases in their scores when their pre-training data set was 
modified, which means that the one item in the scale that they did answer, they answered very positively. 
Adding the sample averages to these very high scores appreciably raised the total scores for this scale. 
The fact that his group would both answer positively and skip questions on this topic is either a 
coincidence or an unknown measurement artifact affecting that one scale. 
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TABLE 3 

Intercorrelations among Pre-training and Post-training Survey Scales 



Pre-Training: 

Delegating Command Responsibility 
Pre-Training; 

Communication & Cooperation 

Pre-Training: 
Managing Stress Effects 

P re -Training: 
Assertiveness 

Pre-Training Control: 
Goal Sharing within Groud 

Pre-Training Control: 
Goal Sharing with Others 

Post-Training: 

Delegating Command Responsibility 

Post-Training: 

Communication & Cooperation 

Post-Training: 
Managing Stress Effects 

Post-Training: 

Assertiveness 

Post-Training Control: 
Goal Sharing within Group 

Post-Training Control: 
Goal Sharing with Others 

Pre-Training: 

Del. Command Resp. 

— j 












.31 j - 











Pre-Training: 

Mng. Stress Effects 

SO | .39 

- 










Pre-Training: 

Assertiveness 

.28 | -.01 

.10 

— 









Pre-Training Control Item: 
Goal Sharing w/Gp. 

.01 j .12 

-.04 

.01 

- 








Pre-Training Control Item: 
Goal Shr w/Others 

.05 \ .08 


-.01 

.74 

— 







Post-Training: 

Del. Command Resp. 


.22 

.26 



- 






Post-Training: 
Communic & Coord. 

-.03 1 .38 

.03 

-.02 

.13 

.08 

0.0 

j - 





Post-T raining: 

Mng. Stress Effects 

.23 j .04 

.48 

.11 

0.0 


.35 

j .08 

- 




Post-Training: 

Assertiveness 

.18 \ -.02 

.06 


.02 

0.0 

.28 

j -.04 

.11 

-- 



Post-Training Control Item: 
Goal Sharing w/Gp. 

-.14 \ .11 

-.09 

-. 06 



-.12 

| .21 

| -.10 

-.03 



Post-Training Control item: 
Goal Shr w/Others 

-.15 | .03 



.24 

.23 

-.18 

j .11 

j ~ 12 

| -.09 

.47 

1 

1 n = 1,822 j 


The pattern of intercorrelations in Table 3 shows that the best predictor of later 
attitudes are the same attitudes at the earlier time period. The pattern also shows the two 
control scales are more related to themselves and to one another than they are related to 
the CRM attitude scales. These results demonstrate that the four CRM attitudes are 
largely independent of one another (a desirable condition), and that they are also little 
related to the two control scales. If anything, there is a slight tendency for the CRM 
scales to be inversely (if only modestly) related to the control scales. That means there is 
a slight, but believable, tendency for scale scores on the goal sharing scales to decrease as 
scores on the CRM attitudes increase. The patterns of intercorrelations found in Table 3 
were replicated in intercorrelations between the Post-training survey and the 2-month, 6- 
month, and 12-month Follow-up surveys. 


14 

















































RESULTS SECTION 1: 


COMPARISON OF ATTITUDES 

All available survey responses are used for the initial comparisons to follow. 

Comparison Of This Company’s Maintenance CRM With Other Companies’ 

The Technical Operations managers and staff professionals in the present company 
were very enthusiastic about this CRM program after they had experienced it. This is 
clear examining their post-training responses to two questions, dealing with their general 
reactions to the CRM program, and comparing them with answers to the same questions 
(also following CRM training) in another Maintenance Department and in Flight 
Operations in other companies. One of these questions asked, “How much will the 
[CRM] training change your behavior on the job?” ( cf . , Appendix Bl, Q28). The other 
question asked, “How useful will the [CRM] training be for others? {cf., Appendix Bl, 
Q27). Figures 1 and 2 present comparisons for the two questions, respectively. 

_____ 

"Is The Training Going to Change Your Behavior On 
The Job?" (Post-Training Survey) 



"No "Slight "Moderate "Large 

Change" Change" Change" Change" 


! H Typical Flight Ops Results E3 Maint.Ops Other Company 
| M Maint.Ops This Company 


X 2 =16.8, df=6, p< OOP 


Figure 1 compares the post-training survey respondents from the present study, on 
the amount of predicted behavior change, with post-training answers to the same question 
from a typical Flight Operations sample (Helmreich, 1989), and from another airline’s 
Maintenance department sample (Taggart, 1990). The present Technical Operation’s 
organization shows a very high level of predicted behavior change (nearly 90% say either 
“moderate” or “large change.” The other Maintenance organization is also very high on 
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the question (over 80% say “moderate” or “large change”), while the typical Flight 
Operations sample is considerably lower on this item, with only 30% predicting 
“moderate” or “large change.” Both maintenance groups are seen to regard CRM training 
as having a very strong potential for personal change. 

FIGURE 2 

How Useful Will This CRM Training Be For Others? 



"Waste of Time” "Slightly Useful" "Somewhat Useful" "Very Useful" "Extremely Useful" 


■ Flight Ops Company A E3 Flight Ops Company B 

■ Maint. Ops Company C El Maint. Ops This Company 


X 2 =25.4, df=12, p<000 


Figure 2 reveals great enthusiasm for CRM by both flight operations and 
maintenance participants immediately following their training. Nearly two-thirds of both 
flight operations samples, shown here, report feeling that the CRM training was “very” or 
“extremely useful” for others. Those Flight Operations samples (Helmreich, 1989) show a 
strong belief in CRM training, but the results to the same question using two maintenance 
samples are even stronger. Ninety percent of Taggart’s maintenance respondents, and a 
similar proportion from the present sample, reported that their CRM training was either 
“very useful” or “extremely useful.” 


Comparison Of This Company’s Maintenance CRM Over Time 

Attitude change from pre-training to post-training surveys. Results using all 
respondents who completed the CRM/TOQ immediately before and immediately after 
training demonstrated that many of the intended effects on participants’ attitudes were 
achieved. Figure 3 presents the pre, post, 2, 6, and 12 month comparisons for the four 
attitude scales using all responses. 
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FIGURE 3 
CRM Attitude Scales 
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□ Pre-Training (n=2056) 

■ Post-Training (n=2053) 

□ 2-Mo Follow-up (n=810) 
E36-Mo Follow-up (n=722) 

■ 12-Mo Follow-up (n=540) 


Shanng Communication Managing Assertiveness 


Command & Cooperation Stress Effects 

Responsibility 


The attitude index scores for "Sharing Command Responsibility," "Communication & 
Coordination," and "Managing Stressor Effects," all increased immediately following 
training. A Multiple Analysis of Variance (MANOVA) “F” was used to test the 
differences among same-scale values for the pre- and post-training surveys and established 
that the pre-post differences noted were statistically significant (F=6.77, df = 2156, 4; 
p<000). "Willingness to Voice Disagreement," a measure of assertiveness and a central 
topic of the training, did not increase significantly immediately after the training. 

Attitude change in follow-up surveys. Once again the MANOVA “F” statistic was 
used to test the differences among the same-scale values for the four surveys taken 
subsequent to the training. With one notable exception, there were no significant 
differences found among the post-training, two, six and 12-month surveys. That 
exception is the assertiveness scale, “Willingness to Voice Disagreement,” which increased 
significantly between the post and 2-month surveys (“t”= 2.16, df = 2,158, p<04). All 
four attitude scale scores remained high thereafter over the remaining period. 

Change in the Control Variables for the Pre-Post and Follow-up surveys. The two 
"Goal Sharing" measures were also compared over the same time period as the four 
attitude scales. The goal sharing scales were not expected to change as a result of the 
CRM training. Although goal setting and attainment are important aspects of maintenance 
work, the training was not designed to directly influence either attitudes or perceptions 
about that aspect of management. Figure 4 presents the pre, post, 2, 6, and 12 month 
comparisons for the two goal sharing scales using all responses. The MANOVA “F” test 
was applied to these data and was found not to be statistically significant (F=1.29, 
p>.05,df= 2156,4). 
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FIGURE 4 

Goal Attainment (Control Scales) 
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B Pre-Training (n=2056) 

■ Post-Training (n=2053) 

□ 2-Mo Follow-up (n=810) 
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B 1 2-Mo Follow-up (n=540) 


Matching Questionnaire Responses Over Time 

Matched questionnaires and sample size. For the remainder of the analyses in this 
report the results for the CRM/TOQ will be reported in terms of the largest number of 
individual surveys received after training that could be matched, by I D. code, to their 
pre-training counterpart. 

CRM training participants were asked to include a confidential identification 
number on the top of their pre-training questionnaire and to write that number on the front 
page of their training notebook. They were encouraged to invent a six-digit number 
known only to them to ensure complete confidentiality. The participants were alerted that 
they would be asked to use the number again on a similar questionnaire at the completion 
of the training, as well as three more questionnaires which would be mailed to them in the 
months to follow. This I.D. number, it was explained, would be used over the times 
sampled to compare answers respondent by respondent rather than merely by groups over 
the four time periods. Participants were assured that the numbers they chose were not 
intended to identify or single out individuals, but merely to match earlier answers on the 
questionnaires with later ones. They were also told that this questionnaire, and all others 
to follow, would be sent directly to the University for processing, tabulating, and 
summarization. 

Rationale for the “matched respondents” sample Using a matched sample insures 
the comparisons made over time are made with the same core group of people, which 
provides added validity to the time-lagged results obtained. One practical cost of this 
matching is that the size of the matched sample for each of the three follow-up surveys is 
reduced, over the total, by the number of respondents who did not include a recognizable 
code with their completed questionnaire. The benefit of using such a sample, however 
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clearly outweighs the penalty of a reduced sample size. Using matched respondents over 
time guarantees that a work group’s average attitude scores, compared over time, will be 
drawn from the same sample There need be no concern that differences found over time 
are the result of comparing respondents who are incomparable with one another. 

Table 4 displays the number (and percentage) of respondents whose later answers 
could be matched with their answers to the previous questionnaires. For the analyses 
included in this report the sample used is of respondents whose subsequent answers were 
matched to their pre-training responses. Table 4 shows that sample, matched to their pre- 
training responses (shown in the upper row in Table 4), to include between 74% and 88% 
of the total responses available. 


TABLE 4 

Numbers of Questionnaires Matched with Previous Surveys 



Post-training 

Survey (n=2,053) 

2 Month Follow- 
up Survey 
(n=810) 

6 Month Follow- 
up Survey 
(n=722) 

12 Month 

Follow-up 
Survey (n=540) 

Matched with 
the Pre-training 

survey (n=2,056) 

1 ,822 (88%) 

598 (74%) 

548 (76%) 

415 (77%) 

Matched with 
Post-training 

Survey (n=2,053) 

- 

561 (69%) 

520 (72%) 

387 (72%) 


Testing for differences among the matched samples, over time. The demographic 
characteristics for the post-training sample, and the three follow-up samples were 
compared with one another, in order to test for possible differences in the composition of 
the four groups. Table 5 presents those comparisons for age, occupation, department, 
education and prior experience. 
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TABLE 


Demographic Comparisons Among the Four After-Training Surveys 


Demographic 

Item 

Post-tng. 

Survey 

2 Month 
Survey 

6 Month 
Survey 

12 Month 
Survey 

Statistical 

Test 

Significance 

Level 

Job Title (9) 

Engineers 

6.6% 

8.8% 

9.3% 

12.4% 

X 2 =82.4 

.000 

Inspectors 

9.6% 

3.8% 

3.5% 

1.8% 

df=24 


Department (8) 

Engineering 

7.1% 

9 . 3 % 

10.8% 

13.1% 

X 2 = 51.0 

.000 

Qua!. Cont. 

11.3% 

7.3% 

6.7% 

6.6% 

df =21 


Location (City 
= 14) 

n/a 

n/a 

n/a 

n/a 

X 2 = 28.6 

df=39 

ns 

Mean Age in 
years 

42.0 

43.0 

42.5 

43.9 

"F” = 3.73 
df=3, 3220 

.001 

Education 

Mean years 
College 

5.7 

5.2 

6.3 

5.7 

“F" = .29 
df=3 f 3196 

ns 

Mean years 
Trade Sch. 

5.1 

4.4 

5.5 

4.9 

“F" =.32 
df=3, 3242 

ns 

Military Exp., 
Mean Years 

6.1 

5.7 

6.5 

6.3 

“F" = .18 
df=3, 3327 

ns 


Age, years of education, and years experience were tested using the ANOVA “F,” while 
the differences among the four survey samples on occupation, and department (being 
categorical or “nominal scale” data) were examined using the “chi-square” (x 2 ) test. Of all 
the tests performed for these data, only age, occupation and department were found to 
show significant differences among the four time samples. Not unexpectedly, the average 
age of the matched sample increases from 42 years at the time of training, to almost 44 
years at the last follow-up survey. Because many 12 months surveys were sent out late 
and delays were seen in their return, this increased age doubtless reflects the mere passage 
of time. Closer examination of the results for occupation revealed that of all the 
occupations a disproportionately large number of Engineers, and small number of Aircraft 
Inspectors, are found in the 12 month sample when compared with those occupations' in 
the post-training survey. Likewise, for the test of differences among departments show 
that Engineering Department and Quality Control (the Inspectors' department) to show 
disproportionately higher and lower numbers, respectively, at 12 months following their 
training. Given that Engineers and Inspectors, together, account for only 16-18% of the 
total sample of matched respondents, the effect of their over and under-sampling at 12 
months is unlikely to affect the overall results reported below. 

Number of work units represented in the “matched sample.” The total of all 1,822 
matched respondents in the post-test sample, includes the members of over 50 
organizational units drawn from all parts of Technical Operations: (Supply, Quality, Base 
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Maintenance, Line Maintenance, Shops, Stores, Materials, Engineering, Planning, and 
Administration). The total numbers of units in each analysis vary depending on the 
specific performance indicator, because not all the work units are measured for the same 
performance. These differences are described in the following section. 


RESULTS SECTION 2: 

TRENDS IN MAINTENANCE PERFORMANCE 
BEFORE AND AFTER THE ONSET OF CRM TRAINING 


Performance Data Described 

Maintenance managers in the company collected performance data in abundance. 
Table 6 presents the five measures used as end-result criteria in the present study. 


TABLE 6 


Performance Measures Selected 1 

For Test 

PERFORMANCE 

MEASURES 

MAXIMUM WORK UNITS 
MEASURED 

RATED SENSITIVITY 
(1 = most sensitive to trng) 


SAFETY 


Aircraft Safety 
(number of maintenance- 
related ground damage 
incidents per flight) 

38 

1 

Occupational Safety 

55 

1 

(lost time injuries, per 1000 



hours) 




DEPENDABILITY 


On-time Departures 
(% departures w/in 5 minutes 
of schedule) 

31 

1 

On-time Maintenance 
(delays due to late from 
planned maintenance, per 100 
scheduled aircraft) 

35 

1 

EFFICIENCY 

Contained Overtime Cost 

55 

2 


(% overtime paid to total wage 
bill) 


Three conditions were met in order to include these measures in the work-unit analysis 
reported here. First (and obviously) the performance measures need to be available by 
work unit (i.e., line station, shop, warehouse, etc ), and not just by department or function. 
Second the measures must be ones that people in the work unit can affect by their actions, 
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or can effectively adapt to; and not merely ones that are conveniently assigned to a unit, 
but for which little can be done. The third condition applied was that the measures not be 
directly related to (or completely determined by) other measures in the set. The five 
performance measures used in the present analysis are included in the company's 
categories of "safety," "dependability," and "efficiency." "Quality" measures used by the 
company are available only for the whole department. Because they cannot be applied to 
the specific maintenance units, quality measure are not included in this present analysis. 

The trainers and administrators of the CRM course evaluated all the available 
performance measures and predicted which of them would be more sensitive to effects of 
CRM training. Their conclusions were that four measures would be the most readily 
improved by the training. These four performance measures included the two safety items 
“ground damage” and “days lost to occupational injury;” and two dependability measures, 
“flight departures within 5 minutes,” and “delays in ‘scheduled work’ (e.g., overnight 
repairs, and overhaul) due to “maintenance error”. Their lower ranked performance 
measures in responsiveness to training included the efficiency measure of “overtime paid.” 


Performance Rates and Trends Before and After the CRM Training Began. 

The "before" and "after" scores for the five performance measures were plotted, and their 
trends compared over the time period for which measures were available. 

Measures of Safety Performance. Safety of Aircraft and Personal Safety. Figure 5 
presents before-and after-training comparisons of number of Ground-Damage incidents for 
38 work units. 


FIGURE 5 

Aircraft Safety (Ground Damage) 



-♦ PreTraining Safety 

■B PostTraining Safety 

" " PreTraining Trend 
PostTraining Trend 
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Figure 5 shows a strengthening of the trend for the decreased number of ground 
damage incidents (measured in incidents per flight), for the 30 months (for which there 
were data available) after the CRM training began, when compared with the six months 
before. The reduced rate for ground damage incidents appears to be related in time to the 
CRM training. 


FIGURE 6 

Occupational Safety (Lost time Injuries) 



Figure 6 shows the rate of lost time injuries, per 1000 hours worked, for 55 work 
units.. In Figure 6 we see a dramatic shift in trend for occupational injury rate comparing 
the six months of pre-training performance with the period afterwards. Closer 
examination of Figure 6 also shows that, except for the unusual (and unexplained) "spike" 
in the fourth and fifth months after the onset of training, the injury rate remained at a low 
level for a year and a half after CRM training was introduced. Following January, 1993, 
the injury rate begins to edge up slightly and then it spikes in September 1993 and again in 
January 1994. Both of those spikes are caused by unusually high rates from individual 
maintenance stations which had been notified that they were scheduled to close. A 
reasonable assumption is that the impending closures affected mechanics' morale which in 
turn affected their attention to personal safety. Results from Figure 6 thus suggest that the 
CRM training caused behaviors leading to greater safety for at least 18 months, until other 
events (e g., site closures and reductions in force) overwhelmed the training's overall 
positive effects. 
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Measures of Dependability. Figure 7 presents before- and after-training 
comparisons for “Departures within 5 minutes of schedule” for the 3 1 line maintenance 
stations for which this measure is relevant. 


FIGURE 7 


Dependability: (On-time Departures) 



+ — Depend/Pre — H — Depend/Post" " " PreT raining T rend ' PostT raining T rend 


Figure 7 demonstrates that line maintenance responsibility for on-time departures 
continues to steadily improve both before and after the training began. Although the 
performance was trending up in the period preceding the training, this chart shows a 
higher level of performance, overall, at the end of the second year following the onset of 
CRM training. 
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Figure 8 shows before and after comparisons for the other indicator of 
dependability, “On-time planned maintenance” — measured by the number of aircraft 
delays due to late from scheduled (or planned) maintenance — for 35 work units 
(including line and heavy maintenance stations). 


FIGURE 8 

Dependability: Late from Planned Maintenance 



& - Delays, Pre-training — 

a — Delays, Post-training 

- - - Pre-training Trend 

KOST-traming i reno 


Although the overall rate for "On-time maintenance” (i.e., delays due to being late 
from scheduled maintenance declining) continually improved over the period measured, no 
appreciable change in that rate is noted in the years following the onset of CRM training, 
when compared with the period before the training began. 
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Measure of Efficiency. Figure 9 shows the pre-post comparisons for the 
percentage of hours of overtime charged for the 55 work units measured. 


FIGURE 9 

Efficiency: Overtime Costs Contained 
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Figure 9 shows that the trend of overtime expenses for the 16 months following the onset 
of CRM training continues the rate of improvement, but that trend then reverses from 
December, 1992 for the remaining 16 months for which we have data. The initial 
improved cost performance appears coincident with the training, and may be affected by it. 
An alternative explanation is that these rates were more affected by top management 
policy in 1991 and 1992, which at first sustained tight controls on overtime expenses, and 
then undertook a labor force reduction which necessarily increased hours for the 
remaining staff. Further analyses below examine this possible effect of changed policy and 
of CRM-related skills used in coping with it. 

RESULTS SECTION 3: 

Testing Relationships Between Attitudes and Performance 

The data sets used in the attitude-performance analyses. There were five data sets 
used for the attitude-performance analyses. By the completion of the current phase of the 
CRM training, in March, 1994, 1,822 of the 2,199 participants attending one of the CRM 
workshops had completed questionnaires such that they could be included in the “matched 
samples” of the pre-, and post-training surveys as well as being the basis of the matched 2, 
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6, and 12 month follow-up surveys. Of those 1,822, 1,692 participants had also provided 
valid departmental identification and, thus, could be included in one of the five data sets. 
The first of those data sets was comprised of 395 participants who were employed in the 
Technical Operations department’s Line Maintenance division. Attitudes from those 395 
Line Maintenance participants in 3 1 work units are tested against the Line Maintenance 
measure of “On-time departures .” For the second data set, Base Maintenance division 
added another 266 participants (in 4 hangar locations), which together with Line 
Maintenance participants provides a total of 661 training participants (in 35 work units) 
for the source of attitude data examined in relationship with the maintenance measure 
“ On-time maintenance .” The third data set added another 477 (in three more city-site 
locations) from the Technical Operations department’s direct support divisions, of Quality 
Control (n=192), Prime Shops (n=l 1 1), and Materials Service (n=174), to test the 
performance measure Aircraft safety. ” That third data set is composed of 1, 138 
maintenance and support personnel in 38 work units. For the fourth and fifth data sets, 
participants in the remaining Technical Operations units for Planning (n=194), Engineering 
(n=120), and Administration (n=240) were added to the data sets above for a total sample 
of 1,692 participants in 55 work units. These samples were used to test attitudes against 
the performance measures of “Occupational safety, “ and “ Overtime costs contained. ” 

The correlation statistics used in the present analysis. The relationships between 
the attitude indices and the performance measures, summarized in Tables 7-1 1 below, 
were calculated using the Spearman Rank-order Correlation statistic ("Rho," or p). The 
use of Rho is advisable in this case because the distributions of these five performance 
measures narrow the choices to non-parametric statistical tests for optimal analytic power 
(Taylor, 1991b). The measured results of four of the five performance measures are 
“improving” when they decline numerically (i.e., the absence of delays due to late from 
planned maintenance, the absence of occupational injuries, the absence of ground damage 
incidents, and reduction of overtime costs). To simplify the presentation of findings all 
results in Tables 7-11 are described as postive coefficients when the original correlations 
are in the expected direction (i.e., favorable attitudes equal better performance). 

Longitudinal relationships between performance and attitudes. A part of the 
following analysis examines the potential impact of performance prior to the CRM training 
on participants’ subsequent attitudes. The more central part of the evaluation of CRM 
training however, is the analysis of those attitudes' effects on subsequent maintenance 
performance. We are interested mainly in the latter analysis — the CRM training's longer- 
term effects on attitudes, and their lagged effect on performance following the training. 

In Tables 7-1 1, to follow, the larger percentages, of 1/3 (33%) or greater, are 
boldfaced as a visual aid. 

Correlations Without Training Effects 

The relationships between the pre-training questionnaire scales and maintenance 
unit performance measures. The first associations examined here are the four pre-training 
attitude scales (measured at the beginning of each training session) with the five 
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maintenance performance indicators. Table 7 contains the percentages for statistically 
significant correlations between each of the four attitudes and each of the five performance 
indicators. 


TABLE 7 

Percentage of Significant Correlations 
Between Pre-training Attitudes and Performance 


Pro-Training Attitude Scales 


Sharing Command 

Communication & 

Managing Stress 

Assertive- 

Responsibility 

Coordination 

Effects 

ness 


Dependability 


Departures within S min of schedule 


1 3 months before training 

40 

40 

_ 1 

20 

26 months after training 

33 v 

__40_ 

ii 1 

IS 


On-time maintenance 


1 7 months before training 

6 

39 

0 

6 

27 months after training 

6 

40 

7 

15 


Safety 


Occupational Safety 


5 months before training 

40 

1 40 

20 

0 

33 months after training 

6 

( 68 

1 3 

24 


Aircraft Safety 


5 months before training 

0 

40 

0 

20 

28 months after training 

6 

_ 13 

10 

3 


Efficiency 
Overtime paid 


5 months before training 

0 

I 20 

0 

0 

33 months after training 

42 

l ii 

15 

21 


Table 7 shows that the pre- training survey attitude scale “Communication & 
Coordination” has a large number of statistically significant relationships with four of the 
five pre-training performance indicators. That means that good performance before the 
CRM training began was quite widely related to positive pre-training attitudes about 
communication & coordination. That pre-training scale measuring communication & 
coordination attitudes was widely related to post-training dependability and occupational 
safety as well. Taken together, these results suggest that although communication and 
coordination is an important value to managers and professionals in the most effective 
units, CRM training may not be a crucial intervention for this effect. This communication 
and coordination scale had a remarkably high initial mean score ( cf . , Figure 3), which 
suggests the possibility that the scale may contain a “social desirability” bias. 
“Communication and cooperation” is also quite a general concept and can have vague or 
multiple meanings. It is also a somewhat “passive concept,” because its constituent 
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questions have fewer specific behavioral references than the other more specific attitudes 
measured with the CRM/TOQ. 


FIGURE 10 

Correlations between: Pre-Training Attitude "Communication 
& Coordination," and Selected Performance Indicators 


Training 



B Aircraft Safety (n=38) ■ Occupational Safety (n=55) □ On Time MX (n=35) E On Time Departures (n=31) 



Figure 10 plots all of the individual correlations between “communication & 
cooperation and the four performance indicators most related to it. Figure 10 reveals a 
large number of months in which the correlations between that attitude and the four 
performance measures are as high as .40. Over time some of those correlations, 
particularly for aircraft safety and on-time maintenance, decline in size. It was reported in 
Table 7 that “communication & coordination” were related to “aircraft safety” in 40% of 
the months measured before training began in June, 1991. To illustrate this, in Figure 10, 
aircraft safety is shown as the first profile of the four. Two of the five correlations (40%) 
before the training began in June 1991 are greater than .25, which is statistically significant 
at the p=.05 level of confidence for a sample of 38. That profile of relationships between 
communication & coordination and aircraft safety demonstrates than the correlations are 
higher and proportionately more numerous before the training began, than afterwards. 


Table 7 also reports that another pre-training attitude with many significant 
correlations with performance is “Sharing Command Responsibility.” This attitude scale is 
seen to be frequently related to line station dependability (“departures within 5 minutes of 
schedule”) both before and after training began. This pre-training attitude about 
delegation is also related to containing subsequent overtime costs. Figure 1 1 displays all 
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the correlations between the pre-training attitude toward sharing responsibility, and 
overtime costs. 


FIGURE 11 

Correlations between: PreTraining Attitude about Delegation, 
and Overtime Costs Contained 


Training 


, Trendline 



Figure 1 1 is a more simple two-dimensional presentation than Figure 10. This 
simplicity allows for the inclusion of a solid line showing the level of significance (p = 22, 
p< 5%) for this performance measure using the 55 work units available for this series, and 
a dotted trendline (calculated by the least-squares regression method) for the correlation 
coefficients over time. Figure 1 1 shows that of the 33 months of overtime measured after 
training began, 14 of them (42%) had correlations above the significance level. 
Furthermore, the trendline in Figure 1 1 shows an upward slope which confirms that the 
correlations between pre-training feelings about participation (“sharing responsibility”) and 
subsequent coping with restrictive overtime policy become more strongly related over the 
period, June 1991 to March 1994. The results in Figure 1 1 clearly show that managers 
and staff professionals who hold more participative values before they attended training 
are subsequently successful in coping with overtime costs. Whether the CRM training can 
impact to further increase that value will be tested below. 

Correlations Showing Training Effects 

The remaining Tables, 8 through 11, report the proportion of relationships 
between the questionnaire scales obtained after the CRM training and maintenance unit 
performance measures. 

Table 8 displays the percent of significant correlations between attitudes obtained 
from the post-training survey and the five performance items. 
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TABLE 8 

Percentage of Significant Correlations 
Between Post-training Attitudes and Performance 


Post-Training Attitude Scales 


Sharing Command 

Communication 6 

Managing Stress 

Assertive- 

Responsibifity 

Coordination 

Effects 

ness 


Dependability 


Departures within 5 min off schedule 


1 3 months before training 

0 

58 

17 

......J........ 

17 

26 months after training 

7 

29 

17 

1 

4 


On-time maintenance 


1 7 months before training 

6 

53 

41 

1. 24 

27 months after training 

7 

33 

37 

i 22 


Safety 


Occupational Safety 


5 months before training I 

20 | 

20 

1 p 

0 

33 months after training | 

9 1 

3 

.1 ? 

18 


Aircraft Safety 


5 months before training 

40 j 

0 

20 

0 

28 months after training 

3 

9 

3 

3 


Efficiency 


Overtime paid . 

5 months before training | 40 | 0 0 | 0 

33 months after training ( 18 j 3 12 [ 3 


Improved attitude about “sharing command responsibility” is seen in Table 8 to be 
related to preceding months of good aircraft safety (lower damage) and lower overtime 
costs. These results suggest that the CRM training’s immediate positive effect on 
attitudes about participation cannot be credited with subsequent performance. Table 8 
also shows that the post-training attitudes about “communication & coordination,” and the 
“value of stress management,” are both very frequently related to on-time scheduled 
maintenance performance both prior to and after the training began. As noted above, the 
attitude scale measuring “communication & coordination,” may be subject to some “social 
desirability” bias, but it is apparently also sensitive to the effects of the training. Because 
the attitude about stress management is lower overall before the CRM training, and the 
training specifically targeted improving stress management, it seems even more likely that 
the training has a direct effect on that attitude and can be credited with subsequent 
performance related to it. Thus, the number of significant correlations between stress 
management and performance for the months after the training began, are likely to be the 
result of CRM training. 
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Figure 12 displays all of the correlations between on-time scheduled maintenance 
performance and the two post training attitudes, “communication and coordination,” and 
“stress managment,” which are reported in Table 8. Figure 12 reveals that the pattern for 
correlations between the two attitudes, and pre and post training performance, is similar 
although communication & coordination has higher correlations before the onset of 
training, with on-schedule maintenance, than does stress management. 
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The relationships between the follow-up surveys and maintenance unit 
performance measures. Tables 9, 10 and 1 1 display the percent of significant correlations 
between attitudes obtained from the 2-month, 6-month, and 12-month follow-up surveys, 
respectively, and the five performance items. 


TABLE 9 

Percentage of Significant Correlations 
Between 2-Month Follow-up Attitudes and Performance 


2 Month Attitude Scales 


Sharing Command 

Communication & 

Managing Stress 

Assertive- 


Coordination 

Effects 

ness 


Dependability 


Departures within 5 min of schedule 


1 3 months before training 1 

75 

J 8 


17 

1 33 

26 months after training ] 

52 

( 11 


7 

.1 5.2 


On-time maintenance 


17 months before training 

65 

0 


0 


0 

27 months after training 

78 

4 


4 


11 


Safety 


Occupational Safety 


r ——j. r 

5 months before training j 

20 

20 

1 0 

1 too 

33 months after training | 

32 

3 

.1 9 

| 62 


Aircraft Safety 


5 months before training 

40 

| 20 

20 

20 

28 months after training 

29 

( 4 

14 

21 


Efficiency 


Overtime paid 


5 months before training 

0 

20 

20 

1 

20 

33 months after training 

9 

3 

21 

I 

3 


T able 9 displays the proportion of significant correlations between maintenance 
unit performance and attitudes from the follow-up survey obtained 2-months after training 
for those respondents with questionnaires matched to their pre-training survey. The more 
passive belief about “communication & coordination” was not found related to subsequent 
maintenance performance in this or in any later survey. Examination of these 2-month 
follow up results, reveals stronger relationships with performance, using attitudes about 
“sharing command responsibility,” and the value of “assertiveness,” than the immediate 
post-training survey showed. Both “sharing command responsibility,” and “assertiveness,” 
are "active" beliefs, the behavioral skills for which were emphasized in the training; and 
which require continued behavioral support, following the training, to reinforce them. 
Sharing responsibility, measured 2 months after training, shows a great many months of 
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significant correlations to post training performance for all dependability and safety 
indicators. 

Figure 13 displays all of these correlations for the “command delegation” scale. 
Note how high many of the correlations are — particularly with the two dependability 
scales (on-time departures, and on-schedule maintenance). 
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It is also reported in Table 9 that assertiveness is related to many (mostly 
subsequent) months of on-time departures, and occupational safety. Figure 14 displays 
these correlations between “assertiveness” two months after training, and performance. 


FIGURE 14 



| □ Occupational Safety (n=55) E3 On-time Departures (n=31 ) 


Overall, the results from this two-month follow-up survey are remarkably strong 
and suggest that lessons learned from the CRM training have consolidated by two months 
afterward and in the more "active" ways, especially in speaking up assertively. 
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Table 10 displays the percentage of significant correlations between the attitudes 
measured six months after training and all available months of performance results. 


TABLE 10 

Percentage of Significant Correlations 
Between 6-Month Follow-up Attitudes and Performance 


6 Month Attitude Scales 


Sharing Command 
Responsibility 


Communication & 
Coordination 


Managing Stress Assertive- 

Effects ness 


Dependability 


Departures within 5 min of schedule 


1 3 months before training 

16 

42 

0 | 0 

26 months after training ] 

4 

26 

6 | 6 


On-time maintenance 


1 7 months before training 

18 

18 

0 

ZIEZ 

6 

27 months after training 

19 

0 

4 

1 

4 


Safety 


Occupational Safety 


5 months before training 

40 

80 

1 40 

0 

33 months after training 

[ ?! | 

18 

i 21 

3 


Aircraft Safety 


5 months before training 

20 

1 

40 


0 

1 0 

28 months after training 

9 

LI 

6 


6 

1 0 


Efficiency 


Overtime paid 


5 months before training | 

0 

20 

T 20 

20 

33 months after training | 

9 

3 

J .21 

3 


These 6 months survey results show a lapse in the effects of CRM training compared with 
the 2-month follow-up survey. They reveal that only prior maintenance performance is 
related to opinions about delegation, communication, and stress management, surveyed six 
months after training. Pretraining occupational safety, was the only performance indicator 
widely correlated to three of the four attitude scales measured six months after training. 
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Figure 1 5 displays all the correlations for occupational safety and the three 
attitudes. Although a number of the correlations in Table 10 are statistically significant, 
most of them are lower in size than those seen for the two month follow-up survey in 
Figures 13 and 14. 


FIGURE 15 

Correlations between: Occupational Safety, & Selected 6 Mo 

Attitudes 



□ Stress Management B Communication & Coordination □ Command Delegation 
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By the 12-month follow-up survey, the training effects have returned to mirror 
aspects of the post and 2-month surveys. Table 1 1 presents the percentage of significant 
correlations between the five performance measures and those attitudes. 


TABLE 11 

Percentage of Significant Correlations 
Between 12-Month Follow-up Attitudes and Performance 


12 Month 

Sharing Command I Communication 6 I Managing Stress I Assertive - 
Responsibility \ Coordination I Effects \ ness 


Dependability 


Departures within S min of schedule 


30 months before survey 

25 

0 

13 


0 

14 months after survey 








On-time maintenance 


25 months before survey 

6 

6 

23 

0 

1 4 months after survey 

11 

6 

52 

o 


Safety 


Occupational Safety 


1 8 months before survey 

35 

12 

18 

76 

21 months after survey 

9 

5 

14 

68 


Aircraft Safety 


1 8 months before survey 

6 

6 

12 1 0 

1 16 months after survey 

0 



0 j 5 


Efficiency 
Overtime paid 


18 months before survey 

24 

6 

1 29 

18 

21 months after survey 

27 

27 

50 

5 


According to Table 1 1, attitudes about the value of assertiveness one year after CRM 
training were related to a great many months of occupational safety both before and after 
that survey was taken. Attitudes, a year after training, toward sharing command decisions 
were (reminiscent of the 2 month results) also related to many months of occupational 
safety, but this time, in the months before the 12-month survey. Similar to the 2 month 
results, assertiveness attitudes were also related to many months of occupational safety 
performance. Figure 16 shows the correlations between occupational safety and the two 
12 month attitudes “sharing command” and “assertiveness.” 
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FIGURE 16 

Correlations between: Occupational Safety, 
and Selected 12 Mo Attitudes 



B Command Delegation 12 months after training □ Assertiveness 12 months after training 


The results in Figure 16 show a reversal of the decline of training results in the six 
month follow-up. That continued improvement of relationships with attitudes after 12 
months reflects further strengthening of the training effects. This suggests that the 
observed diminution of associations using the earlier, six month attitude data (c/. Figure 
15), represents an internal consolidation of training effects rather than their alleviation. In 
effect, maintenance managers and staff professionals recognized, in the months following 
CRM training, that their new knowledge and attitudes about participation was of value 
and that communication, and the effect of practicing the appropriate behaviors, could have 
an effect on their units’ safety. 
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Another, very interesting outcome of the 12-month survey, reported in Table 1 1, 
were the relationships between managing stress effects and the many subsequent months 
of on-schedule maintenance, and overtime cost savings. These results follow June 1992, a 
full year after the onset of CRM training in June 1991. They represent performance for 
both of these very stressful requirements (holding to planned maintenance schedules and 
staying within required overtime levels) for an additional lagged period during which the 
lessons of stress management, learned during training a year earlier, had also consolidated 
for the training participants. Figure 17 presents the correlations between those stressful 
requirements and 12 month attitudes toward stress management. 

FIGURE 17 

Correlations between: 12 Mo Stress Management Attitude, 
and Selected "Stressful” Indicators 



H Overtime Costs Contained (n=55) E On-time Maintenance (n=35) 
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RESULTS SECTION 4: 

SELF-REPORTS OF BEHAVIOR CHANGE 


Written Comments from the Post-Training and Follow-up CRM/TOQ 

Two open-ended questions were asked of the maintenance CRM training 
participants concerning the intended use of the CRM training and its application to the job. 
We considered this type of question as a self-report of the perception of ones' intention 
and behavior change on the job. What follows are the results of the two questions. The 
full sample available through December, 1994 is employed in the following analysis. 

Open-ended question about intended use of CRM training 

Participants were asked to write their responses to the first of these questions 
"How will you use this training on your job?" That question {cf , Appendix Bl, Q IV; 
Appendix B2 Q30) was included on the post-training, two, six and twelve month follow- 
up CRM/TOQ's. Content coding the participants responses to that question resulted in 
the bulk of the responses divided into three categories, including: 1) "Being a better 
listener," 2) "Being more aware of others" and, 3) "Dealing better with others. " These 
three categories represent two types of self-perceived potential behavior changes; passive 
and active. For the first two categories which we classified as "passive" behaviors, 
intended improvements are made within the person such as, "I am aware of other 
personalities and management styles" or "I am more tolerant of others opinions. " 
Conversely, the other category, classified as "active" behaviors, contains intended 
behaviors such as, "communicating better in giving instructions on work assignments," as 
well as "frequently using problem solving activities," "group consensual decision making," 
or actually, "dealing better with others. " 
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Figure 1 8 shows the results of the question for all four measurement waves. In the 
post-training survey, none of the three categories accounts for as much as 15% of those 
answering. In the subsequent waves of measurement in the months following, the two 
"passive" categories show a relatively flat infrequent pattern, while the "active" category 
"deal better with others" substantially increases to 37%. The significance of this pattern 
was tested using the Chi Square test, which revealed a clear difference. Clearly, there is a 
shifting pattern over the four survey periods taken after training from the passive reported 
behavioral intentions to the more active behaviors of working better with others and 
interpersonal interactions. 


FIGURE 18 

"How Will You Use This Training On Your Job?" 
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E3 Better Listener 
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□ Deal Better w/Others 


X 2 =22.3, df=6, p< 001 
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Open ended question about how training was used on the job 

Responses to the question "What changes have you made as a result of the CRM 
training" ( cf , Appendix B2 Q29) were content coded in the same three categories as the 
open ended question above. Figure 19 shows initially higher levels for the two passive 
categories ("better listener" and "be more aware of others") for the 2 month and 6 month 
surveys. By 12 months after their training respondents reported more active behaviors 
The patterns, in Figure 19, over time were statistically significant using the Chi Square 
test. 


FIGURE 19 


"What Changes Have You Made As A Result Of The 
CRM Training?" 

50 -] 


Two Six Twelve 

Month Month Month 

(n=714) (n=470) (n=355) 


El Better Listener 
■ Be More Aware 
□ Deal Better w/Others 



1-14.3, df=4, p<01 


This pattern shift from passive reactions to the increasing use of active behaviors 
indicates that the managers are applying CRM skills and knowledge acquired from the 
training. Furthermore, the preferred behaviors have shifted from those behaviors people 
could do by themselves (e g. , "be a better listener" and "being more aware of others"), to 
behaviors which involve others, such as "having more daily meetings to solve problems" 
and "gathering more opinions" and "getting more feedback from others. " It appears that 
after the training, the managers returned to the work site and began to process the training 
by first applying it to passive behaviors, such as being more aware of different personality 
and management styles, being more tolerant and patient of others' personal opinions. 

Then gradually over time, the participants became more comfortable with using these 
passive behaviors and recognized their positive effects, which then encouraged them to 
initiate more active behaviors. Thus, this increase use of active skills by the managers. 
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several months following the training, confirms that the learned CRM skills were achieved 
and improved interpersonal behaviors occurred. 

Self-reported degree of behavior change 

Shown in Figure 20 are the results of the managers' self-reported responses to the 
question about the effect of training on their behavior ( cf. , Appendix Bl, QIV; Appendix 
B2, Q28). A spike of “post-training” responses is seen for the category "large [behavior] 
change. " The subsequent "moderate" behavior change remains stable over the three 
remaining survey periods following the CRM training. These results suggest that positive 
behavior changes did occur following the CRM training and are likely to be a result of the 
training. Initially, it appears that the managers readily applied the knowledge gained from 
the CRM training to their work environment, then over time having had these behaviors 
already incorporated into their daily work routine, they reported a more moderate level of 
behavior change several months after the training. From the previous results of the 
managers comments, we know that these types of reported behavior changes occurring on 
the job are those of a more active nature. 


FIGURE 20 


Self-Reported Degree of Behavior Change: All 
Surveys 



"No "Slight "Moderate "Large 

Change" Change" Change" Change" 

| E3 Post-Training (n=1 ,773) BTwo Months (n=809) □ Six Months (n=722) □ 1 2 Months (n=540) 


X 2 =25.1 . df=9, p<.002 
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Analysis of Results of Field Interviews and Observations 


Methodology for field interviews and observations 

To validate the contents of the managers' self-reported behaviors as well as to 
expand the base of behavior measures used in the maintenance environment, we conducted 
field interviews and observations of the technical operations over a one year period 
(November 1993-November 1994). We selected maintenance sites based on their 
maintenance performance, geographical location, "organizational culture," and functions 
(e. g. , line stations and base maintenance hangars, and prime shops). Using the 
maintenance performance measures of dependability (departures within 5 minutes and late 
from planned maintenance) and safety (occupational safety and ground damage), we 
classified the stations into three categories of performance, low, medium and high and 
selected several from each category. We determined a representative sample consisting of 
ten U. S. city sites. We conducted individual and confidential interviews with a specified 
sample of maintenance managers who represented several levels of supervisory positions 
and work shifts: Regional and station managers (n=4). Supervisors (n=10). Assistant 
Supervisors (n=16), Technicians (n=6), and Administrative Assistants (n=2). We 
developed an interview protocol for these onsite interviews of maintenance managers 
consisting of questions pertaining to the successful transfer of CRM skills and knowledge 
to actual behaviors demonstrated at the jobsite. Four areas were covered in the field 
interviews: 1) determining what respondents liked and disliked about the training; 2) 
listing the applications of what they learned in their CRM training and how they had been 
using the training differently over time; 3) describing and elaborating their behaviors using 
examples from their work, and 4) describing upper management's support of the CRM 
training program. We observed the technical operations for all three work shifts as well as 
during the shift changes. It was during these shift changes, that we found a high level of 
discussions and interactions among the managers and technicians. These discussions 
focused on procedures involving the assignment of aircraft maintenance work and the 
clarification of work procedures. 

Analysis of the interview data 

We organized the interview data according to the four areas noted above. The 
following section specifically focuses on interview comments illustrating examples of 
passive and active behaviors. Subsequent sections review the interview data dealing with 
likes and dislikes of the CRM training, and the upper management support of the program. 

Application of lessons learned and different use over time 

The respondent’s comments were classified as either examples of passive or active 
behaviors These interview data were then compared with the written comments reviewed 
above. Tables 12 and Table 13 present the consolidated results of both the written 
comments and interview comments for passive and active behaviors, respectively. In these 
tables, we included only those interview comments that were stated by at least 50% of the 
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managers, as well as comments from the survey questionnaires that demonstrated the 
strongest examples of either passive or active behaviors. 


The interview comments shown in Table 12 provide illustrative examples of 
passive behaviors that the managers were using after the CRM training program. These 
comments further support the finding that when the managers returned to the work site 
and began to process the training, they first applied it to passive behaviors. Many of the 
managers said that they were more tolerant of others, were better listeners concerning 
different inputs, and that they would try to "cool down" before they jumped to a 
conclusion. As the managers began to recognize the positive impacts of their attitudes, 
they became more comfortable using these behaviors and were encouraged to initiate 
more active behaviors. 


TABLE 12 

For Passive Behaviors 


WRITTEN COMMENTS* 

INTERVIEW COMMENTS** 

Coded as “Passive Behaviors” 

Coded as “Passive Behaviors” 

"Think before giving my response" 

”1 stop and think when 1 get angry” 
n = 17 

"Control my emotions" 

"cool down" 
n = 15 

"1 am more tolerant of others opinions" 

"More tolerant of others" n = 25 

"1 understand now that there are two sides of a 
story" 

"1 listen to the whole story now” 
n = 26 

"1 am aware of other personalities 
and management styles" 

"aware of different management styles" 
and the variety of personalities" n = 24 

"1 understand how 1 react to stress" 

"1 manage my stress better" 
n = 31 

”1 am not judgmental as 1 used to be" 

”1 don't jump to conclusion and blame 
someone right off" 
n = 15 

1 am a better listener" 

"Better listener to different inputs" 
n = 22 


* From Post-Training,, Two and Six months Follow-up Surveys 
** n = 30, maintenance operations managers interviewed 
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Shown in Table 13 are illustrative examples of active behaviors that were used by 
the managers. Managers commented that understanding others’ opinions and being more 
assertive in expressing their own opinions in meetings allowed everyone to fully 
understand why certain procedures and processes were done. For example, during the 
shift changes managers would insure that everyone clearly understood the current status of 
all aircraft, any urgent issues that needed to be addressed, and potential problems that 
might occur later in the work period. Openly recognizing the high stress conditions that 
would occur in their work situation, the managers expressed that they would communicate 
more effectively with the group by working with the team in understanding everyone's 
point of view. This would generally occur during their daily meetings that were scheduled 
between shifts. After thoroughly discussing potential solutions, the teams would come to 
a collective decision, fully supported by the team Several managers said that slowing 
down the decision process and asking for feedback from their staff was beneficial. It was 
then understood why a certain decision had been made and everyone was part of the 
decision process. Even under stressful conditions, the managers stated that they would 
ask their workers if they understood their instructions and the procedures that they were 
suppose to follow by having them repeat back to them the instructions and why they were 
doing it in a particular manner. Doing this, they said, helped everyone understand what 
was to be done and the best way to get the job done. 

An example of an active behavior engaged in by managers was communicating 
why certain maintenance job procedures and processes were called for. Those processes 
included selecting shift work schedules, overtime bidding, and safety procedures. The 
managers stated that once they asked for more feedback from their workers regarding 
work procedures, they began to receive constructive inputs from them on work issues that 
were important to them. Thus, the managers said that they would either communicate 
better why a particular work process or procedure was to be done or they would change 
the work procedure itself Several times, the managers stated that they would visibly post 
the work process, such as shift bidding, in the break room for all employees to view. 

Then, if there were any specific questions as to why a certain process was taken, the 
managers could explain to the group, during their daily shift work meetings, as to how the 
process was decided and why. Many managers commented that when they did this, the 
problems that were occurring in the workplace seemed to be taken care of because 
everyone was informed. If there were any further questions workers could refer to the 
posted procedures posted in the breakroom. In some instances, the workers would have 
some ideas on how to improve the process since they had the opportunity to study the 
posted procedures. The managers said since their workers were better informed they were 
providing more constructive ideas and sometimes they would incorporate their ideas. 

Thus, a more constructive dialog among the work team would occur enhancing better 
communication among themselves 


47 



j®;TABLE iz 

Comparison of Written and Interview Comments 
For Active Behaviors 


WRITTEN COMMENTS* 

INTERVIEW COMMENTS** 

Coded as “Active Behaviors” 

Coded as “Active Behaviors” 

"We all have different opinions and 
viewpoints--more open minded and 
participative” 

"understanding other's opinions by reiterating 
what they thought they heard" n=26 

1 am more assertive 

"Being more assertive by voicing my opinions in 
meetings or discussions" n=22 

"1 ask the guys more questions now" 

"Continuously working on saying what 1 mean to 
my co-workers as 1 as for feedback on what was 
understood” n=22 

"Expressing the why of the process" 
'The why explained" 

"Telling the workers the WHY behind the 
procedure or process-showing them explicitly 
why and taking the time for them to understand" 
n=27 

"Encouraging others to recognize stress" 

"Recognizing stress with aircraft problems and 
telling everyone we gotta work this one out 
together" n=28 

I "Communicating better under stress" 

"A group decision we all came to-understood 
everyone's point of view-solutions arrived under 
the high stress level" 
n=28 


* From Two, Six and Twelve Months Follow-up Surveys 
** n = 30, maintenance operations managers interviewed 


Likes and Dislikes of the CRM training program 

What participants liked about the training. There was an overwhelming positive 
response from the managers about how much they enjoyed the class and the usefulness of 
the CRM training. Every managers interviewed commented that this training was one of 
the best courses that they had attended at this airline. Not only were they able to apply the 
training to their work environment, several managers said they also used the CRM skills 
at home, allowing them to "practice" their newly attained skills. Over 90% of the 
managers commented that the Strength Development Inventory (SDI) personality style 
exercise was one of the particularly good aspects of the CRM training. Using SDI, the 
participants were given information concerning their behavioral style with the implications 
of that style for human interactions and management practices. Learning ones' behavioral 
style under various workloads and conflict situations, appears to heighten the awareness 
and appreciation of how people will react and manage. The managers said that having 
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understood their behavioral style brought a sense of clarity about the responses of other 
toward them under various leadership and working situations. Also, understanding their 
boss’ behavioral style helped them understand why the boss was managing the work units 
a particularly way. 

Other training modules the managers particularly liked were assertiveness, stress 
management and active listening. Managers reported that that they better understood 
their own behavioral style and their bosses' behavioral style, learning how to become more 
assertive, manage stress better and communicate more effectively under stressful 
conditions was smoother and more productive. 

Another aspect of the CRM training that the managers said they particularly liked 
was the mix of the participants in the class. Being in a class where the trainees were from 
other departments representing different functions and responsibilities, allowed people to 
gain an appreciation of the other mangers job functions, what their constraints and 
problems were, as well as how the outputs of their jobs affected others in the work 
system. There were several group exercises given in the training, allowing for small group 
of participants to work together which provided everyone the opportunity to actively 
experience, hear and discuss the how’s and why’s of arriving at a decision. Having this 
mix of participants drawn from the various line and base maintenance stations provided 
everyone with the opportunity to meet their co-workers and to associate an individual 
with a particular job functions Several managers commented that this allowed them to 
see the "human" side of their work environment. For example, "Now I know who Charlie 
is at the line station and I feel much more comfortable with working with him on solving 
our problems." "It just seems to make the whole process a little easier and sometimes 
under our stressful conditions we laugh about which way our "arrows" are pointing. " (In 
the SDI, each participant plots "arrows" according to their responses to several questions 
of how they would react to a situation under stress and under more normal circumstances. 

Over 95% of the managers commented that they liked the role playing training 
exercise where the roles were defined and scripted and the scene was a believable 
interaction among mechanics, maintenance managers, inspectors and airline pilots. They 
all expressed that they would like to have more role playing exercises where everyone 
would be able to participate and experience the different roles. 

Participants’ dislikes of the training. No manager expressed that they did not like 
the training. The only item of concern, raised by two managers, was that sometimes the 
material presented was a little too theoretical, however this was not perceived as a big 
issue. 

Upper Management’s Support Of The CRM Training 

It was expressed by every CRM training participant that they were pleased that the 
CRM training was a required activity with the complete support of the senior Vice 
President of Technical Operations. This type of support indicated to the maintenance 
managers that the company was very serious about this training and the intended skills and 
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knowledge that was to be acquired. Over 95% of the managers commented that with this 
type of training mandate, it allowed them to not have to struggle to attend the course, or 
justify their subordinates’ time to attend this training program as well. 

All of the assistant supervisors interviewed, reported that this was their only 
occasion to attend a training course which was not technical in nature but was related to 
developing their management and interpersonal skills. 

All mangers commented that they would like to see management support for a 
refresher course to further develop and practice their newly acquired CRM skills. In the 
follow-up CRM training course, it was suggested that there should be more case studies, 
role playing exercises, group decision making exercises and conflict resolutions exercises. 
Furthermore, continuing the practice of having the CRM course participants representing 
the various job functions and roles was emphasized as very important by the managers. 
Several of them also said it would be valuable to offer the CRM course to the mechanics 
and inspectors as well. They would like to see management support the mandatory 
attendance of mechanics to a CRM course. This, they said, would provide a common base 
of working CRM knowledge and skill development among all technical operations 
personnel. 

Several managers said it would be useful if relevant case studies or exercises could 
be provided to them periodically so they could pull together their maintenance team and 
constructively practice their CRM skills. It was suggested that using time in their monthly 
safety meetings might be an appropriate forum, if management would initiate and support 
this type of process. They said it would indicate to the technical operations personnel that 
the continual practice and application of CRM skills is considered important in the 
company. 

Summary, Validation of the self-reported behaviors in the CRM/TOQ 
questionnaires was demonstrated by the results of the field interviews and observations. 
Confirmation of the reported change of behaviors from passive to more active and 
interactive behaviors over time, demonstrated the positive transfer of CRM learned skills 
and knowledge to the work environment and job. Commitment to using the CRM skills 
were shown, over time, by various communication interactions among the technical 
operations managers confirming and validating that CRM skills were incorporated into 
their daily tasks of technical operations. Illustrative examples collected during the 
interviews from several technical operations managers, provided testimonial evidence 
demonstrating and further supporting the success and positive effects that the CRM 
training had on managers' attitudes and behaviors. 
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CONCLUSIONS 


The increased safety, dependability and improved costs associated with positive 
attitudes following the CRM training are strong effects, and they are attributable to that 
training. The impressive number and size of the correlations between post training 
attitudes and performance, together with the longitudinal association with performance 
months after the training are very convincing evidence for the positive effects of CRM 
training. The reported behaviors obtained from open-ended questionnaire items, on-site 
interviews and observations further confirm the real effects those numbers imply. 

Comparisons of managers' attitudes immediately after their training with their pre- 
training attitudes show significant improvement. Improvement took place in attitudes 
about "willingness to share command responsibility," "usefulness of communication & 
coordination," and "recognition that stressors affect decision making. " The same 
attitudes of these managers two-, six-, and 1 2-months later reveal that these favorable 
post-training attitudes remain at those positive levels in the months after the training. The 
fourth attitude scale measured, “willingness to voice disagreement” (a measure of 
assertiveness), although showing no significant change immediately following training, 
improved significantly above the pre-training levels two months after training and it 
remained at that higher level six and 12 months afterwards. The influence of the training 
on all the expected participant attitudes is thus a stable and robust effect and not merely a 
brief "honeymoon” effect of good feelings immediately after the training. 

Other analyses of the data looked at the relationships between the four attitudes 
and maintenance results over several years. In this “time-lagged” or longitudinal 
correlation analysis we tested the effects of the training program on performance. We 
compared performance before the onset of the training, to attitudes in subsequent time 
periods, as well as comparing attitudes with later performance results. We found an 
explainable pattern of significant relationships between the participant’ pre-training and 
post-training attitudes, and between those attitudes and performance. Positive attitudes 
about “assertiveness,” “sharing responsibility,” and “stress management” -- all skills and 
behaviors specifically taught in the CRM training — were more predictive of subsequent 
performance. Positive attitudes about the more passive belief, “communication and 
cooperation,” seemed less sensitive to the CRM training and were less predictive of 
subsequent performance The attitude, communication & coordination, was not found 
related to maintenance performance in any of the follow-up surveys. Positive attitudes 
were apparently initially valuable, as they suggested a sensitivity to more specific 
behaviors and later attitudes about assertiveness, stress management, and delegation. 

Associations between positive post-training attitudes and subsequent performance 
were even stronger when attitudes were measured at 2 months and at 12 months after the 
training. Overall, the survey results from these 2 month and 12 month follow-up surveys 
are remarkably strong and indicate that lessons learned from the CRM training consolidate 
and strengthen in the months afterward, and in more “active” or practical ways. In most 
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cases the strongest relationships were found in the months following the measurement and 
they then trended downward for the remaining period. 

The strong implication is that CRM training for maintenance managers and support 
professionals is effective if the behaviors taught are given time to develop. Once 
competence and early success is achieved consolidation takes hold and reinforces the 
positive attitudes making them more enduring. Simply letting time take its course does 
not assure the continued success of the program. The period between two months and 
one year following initial training should be an ideal time for recurrent training to sharpen 
the skills and to strengthen management resolve. Additionally the motivation and logic to 
extend the CRM training to mechanics is strong among past participants and this diffusion 
should be realized. With diffusion to mechanics, and recurrent training for all participants, 
we should then expect to find that ever increasing maintenance performance would be the 
outcome of the positive attitudes and inspiration resulting from “crew resource 
management. ” 

Analyses of the managers' self-reported behaviors and interview comments confirm 
a shift of behaviors from passive, to more active and interactive behaviors over time. This 
demonstrates the positive transfer of CRM learned skills and knowledge to the work 
environment and job. Commitment to using the CRM skills were shown over time, by the 
various communication interactions among the technical operations managers, further 
supporting and validating that CRM skills were incorporated into their daily tasks of 
technical operations. As the managers became more comfortable with using passive 
behaviors and began to recognize the positive impacts that their attitudes had, they were 
encouraged to initiate additional active behaviors. We observed that at first, the managers 
reported and expressed that they became better listeners. Then over time, they began to 
become more assertive, handled stressful situations better, informing their workers about 
job procedures in scheduled daily shift work meetings, and understanding other's opinions 
better by repeating what they thought they had heard. With this increased use of active 
skills by the managers, the intended CRM skills were learned and achieved and improved 
interpersonal behaviors did occur. 

There is a strong implication that greater reinforcement of this CRM for 
maintenance program would have enhanced the already very encouraging results reported 
here. That reinforcement could take several parallel and separable paths. First greater 
visible support from upper-middle management would confirm company commitment to 
the program. Such visibility would result from department executives and managers 
making a brief appearance at the opening or closing of each training session. Next, our 
results show that recurrant training in active and assertive communication skills would 
benefit participants, especially if that training took place within six months of the initial 
training (and perhaps as soon as two months afterwards). Finally, the advantage of 
including mechanics (Aviation Maintenance Technicians, or AMTs) in the training sessions 
was not lost on many participants, who suggested that this be done. In fact the company 
has recently begun a CRM program for AMTs and we look forward to learning of its 
successes. 
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contributed to accident. 

How can we test our assumptions? 
What could each person have done 
differently? 
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Discuss how to manage norms and prevent 
accidents. 
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APPENDIX B 1 




Seminar Date 


ID CODE 


TECHNICAL OPERATIONS DIVISION 
CCC WORKSHOP 

POST SEMINAR 

Please answer by writing beside each item the number that best 

reflects your personal attitude. Choose the number from the scale 

below. 

*****Scale***** 

1 2 3 4 5 

Disagree Disagree Neutral Agree Agree 

Strongly Slightly Slightly Strongly 

1. Technical Operations team members should avoid 

disagreeing with others. 

2. It is important to avoid negative comments about the 

procedures and techniques of other team members. 

3. Casual, social conversation on the job during periods of 

low workload can improve Technical Operations team 
coordination . 

4. Good communications and team coordination are as 

important as technical proficiency for aircraft safety 
and operational effectiveness. 

5. We should be aware of and sensitive to the personal 

problems of other Technical Operations team members . 

6. The manager, supervisor, or assistant supervisor in 

charge should take hands-on-control and make all 
decisions in emergency and non-standard situations. 

7. The manager, supervisor, or assistant supervisor in 

charge should verbalize plans for procedures or actions 
and should be sure that the information is understood 
and acknowledged by the other Technical Operations 
team members. 

8. Technical Operations team members should not question the 

decisions or actions of the manager, supervisor, or 
assistant supervisor except when they threaten the safety 
of the operation. 
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1 

Disagree 

Strongly 


*****Scale***** 


2 3 4 5 

Disagree Neutral Agree Agree 

Slightly Slightly Strongly 


_9. Even when fatigued, I perform effectively during critical 
phases of work. 

10. Managers, supervisors, and assistant supervisors should 
encourage questions during normal operations and in 
special situations. 

11. There are no circumstances where the subordinate should 
assume control of a project. 

12. A debriefing and critique of procedures and decisions 
after each major task is an important part of developing 
and maintaining effective team coordination. 

13. Overall, successful Technical Operations management is 
primarily a function of the manager's, supervisor's, or 
assistant supervisor's technical proficiency. 

14 . Training is one of the manager's most important 
responsibilities . 

15. Because individuals function less effectively under high 
stress, good team coordination is more important in 
emergency or abnormal situations. 

16. The start-of -shift team briefing is important for safety 
and for effective team management . 

17. Effective team coordination requires each person to take 
into account the personalities of other team members. 

18. The responsibilities of the manager, supervisor, or 
assistant supervisor include coordination between his or 
her work team and other support areas. 

.19. A truly professional manager, supervisor, or assistant 
supervisor can leave personal problems behind. 

20. My decision-making ability is as good in abnormal 
situations as in routine daily operations. 
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*****Scale***** 


1 2 3 4 5 

Disagree Disagree Neutral Agree Agree 

Strongly Slightly Slightly strongly 


In the following questions, "my management group" refers to those 
people who report to the same manager that I do. 


21. I am kept informed by others in my management group about 
the goals and objectives of this organization (e.g., 
cost, quality, service, etc.). 

22. Work goals and priorities are understood and agreed to by 
members of my management group. 


In the following items, "my work group" refers to those people who 
report to me. 


23. Employees in my work group receive detailed feedback 
regarding the organization's performance. 

24. If employees in my work group disagree with the goals and 
priorities that have been established, they feel free to 
raise their concerns with supervision. 

25. Employees in other groups within Technical Operations plan 
and coordinate their activities effectively together with 
people in my work group. 

26. Employees in other groups, departments and divisions 
throughout the company act as if they share many of the 
same organizational goals that we do. 
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TRAINING EXPERIENCE AND EVALUATION 


I* For each of the topic areas or training techniques listed 
below, please rate the value of this aspect of the 
training to you. Rate each item by cooosing the number on 
the scale below which best describes your personal opinion and 
then write the number beside the item. 

1 2 3 4 5 

Waste of Slightly Somewhat Very E x t reme ly 

Time Useful Useful Useful Useful 


Training in interpersonal communications and skills 
Assertiveness 


Conflict resolution 


Stress effects and stress management 

Analysis of personal styles and dimensions of team 
leadership 

Testing assumptions 

Training in skills using role play. 

Case studies of aircraft accidents and incidents 


Norms 


Active Listening 

Overall, how usefull did you find the training? 
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II • Technical Operations ' Crew Coordina tion Concept* training has the 
potential to increase aviation safety and teamwork 
effectiveness. (circle one from list below) 

Disagree Disagree Neutral Agree Agree 
Strongly Slightly Slightly Strongly 


III. How useful will such training be for others? (circle one 
from list below) 

Waste of Slightly Somewhat Very Extremely 

Time Useful Useful Useful Useful 


IV. is the training going to change your behavior on the job? 
(circle one from the list below) 

No Change A Slight A Noderate A Large 

Change Change Change 


V. How will you use this training on your job? 



VI. What aspects of the training were particularly good? 



VII. What do you think could be done to improve the training? 
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Year of birth 

Total years at CAL 

Sex (M or F) 

CURRENT DEPARTMENT 

Line Maintenance 

Base Maintenance 

Quality Control 

Planning 

Shop 

Material Services 

Engineering 

Other 

WORK LOCATION - CITY 

• 

Job Title: 

Years in present position: 

Past experience/training (# of years) : 

Military 

Trade School 

College 

Other Airline 

This completes the questionnaire 
Thanhs for your help. 
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TECHNICAL OPERATIONS DIVISION 
CCC WORKSHOP SURVEY 


Date 


"Six-month Follow-up" Questionnaire 


Please enter the five digit Personal Identification Number 
that you selected at the beginning of the seminar. 

Identification Code 


Now, please answer by writing beside each item the number that best 
reflects your personal attitude. Choose the number from the scale 
below. All data are strictly confidential. 


*****Scale***** 


1 

Disagree 

Strongly 


2 

Disagree 

Slightly 


3 

Neutral 


4 

Agree 

Slightly 


5 

Agree 

Strongly 


1. Technical Operations team members should avoid 
disagreeing with others . 

2 It is important to avoid negative comments about the 
procedures and techniques of other team members. 

3. Casual, social conversation on the job during periods of 
low workload can improve Technical Operations team 
coordination . 


4. Good communications and team coordination are as 

important as technical proficiency for aircraft safety 
and operational effectiveness. 

5 We should be aware of and sensitive to the personal 
problems of other Technical Operations team members. 

6. The manager, supervisor, or assistant supervisor in 
charge should take hands-on-control and make all 
decisions in emergency and non-standard situations. 

7. The manager, supervisor, or assistant supervisor in 
charge should verbalize plans for procedures or actions 
and should be sure that the information is understood 
and acknowledged by the other Technical Operations 
team members. 


PA Of . 
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1 

Disagree 

Strongly 


*****Scale***** 

2 3 4 5 

Disagree Neutral Agree Agree 

Slightly Slightly Strongly 

8 . Technical Operations team members should not question the 

“ decisions or actions of the manager, supervisor, or 

assistant supervisor except when they threaten the safety 
of the operation. 

_9. Even when fatigued, I perform effectively during critical 

~ phases of work. 

10. Managers, supervisors, and assistant supervisors should 
encourage questions during normal operations and in 
special situations. 

11. There are no circumstances where the subordinate should 
assume control of a project. 

12. A debriefing and critique of procedures and decisions 
after each major task is an important part of developing 
and maintaining effective team coordination. 

13. Overall, successful Technical Operations management is 
primarily a function of the manager's, supervisor's, or 
assistant supervisor's technical proficiency. 

14. Training is one of the manager's most important 
responsibilities. 

15. Because individuals function less effectively under high 
stress, good team coordination is more important in 
emergency or abnormal situations. 

16. The start-of -shift team briefing is important for safety 
and for effective team management. 

17. Effective team coordination requires each person to take 
into account the personalities of other team members. 

18. The responsibilities of the manager, supervisor, or 
assistant supervisor include coordination between his or 
her work team and other support areas. 

19 . A truly professional manager, supervisor, or assistant 
supervisor can leave personal problems behind. 

20. My decision-making ability is as good in abnormal 
situations as in routine daily operations. 
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*****Scale***** 


1 

Disagree 

Strongly 


2 

Disagree 

Slightly 


3 

Neutral 


4 

Agree 

Slightly 


5 

Agree 

Strongly 


In the following questions, "my management group" refers to those 
people who report to the same manager that I do. 

21. I am kept informed by others in my management group about 
the goals and objectives of this organization (e.g. , 
cost, quality, service, etc.). 

22. Work goals and priorities are understood and agreed to by 
members of my management group. 


In the following items, "my work group" refers to those people who 
report to me. 


23. Employees in my work group receive detailed feedback 
regarding the organization's performance. 

24. If employees in my work group disagree with the goals and 
priorities that have been established, they feel free to 
raise their concerns with supervision. 

25. Employees in other groups within Technical Operations plan 
and coordinate their activities effectively together with 
people in my work group. 

26. Employees in other groups, departments and divisions 
throughout the company act as if they share many of the 
same organizational goals that we do. 


27. How useful has the CCC training been for others? (circle 
one) 

A Waste Slightly Somewhat Very Extremely 

of Time Useful Useful Useful Useful 

28. How much has the CCC training changed your behavior on 
the job? (circle one) 

No Change A Slight A Moderate A Large 

Change Change Change 
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29 . 


What changes have you 
training? 


made as a result of the CCC 




How will you further use the CCC training in the coming 
months? 





Looking back on it now, what aspects of the training were 
particularly good? 


What do you think could be done to improve CCC training? 





CURRENT DEPARTMENT 


Line Maintenance 
Base Maintenance 
Quality Control 
Planning 
Shop 


Material Services 


Engineering 

Other 


WORK LOCATION - CITY_ 


Job Title : _____ 

Years in present position: 

Past experience/training (# of years): 

Military 
Trade School 
College 
Other Airline 


This conpletes the Questionnaire 
Thanks for your help. 
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